Task-irrelevant semantic relationship between objects and scene influence attentional allocation

Recent behavioral evidence suggests that the semantic relationships between isolated objects can influence attentional allocation, with highly semantically related objects showing an increase in processing efficiency. This semantic influence is present even when it is task-irrelevant (i.e., when semantic information is not central to the task). However, given that objects exist within larger contexts, i.e., scenes, it is critical to understand whether the semantic relationship between a scene and its objects continuously influence attention. Here, we investigated the influence of task-irrelevant scene semantic properties on attentional allocation and the degree to which semantic relationships between scenes and objects interact. Results suggest that task-irrelevant associations between scenes and objects continuously influence attention and that this influence is directly predicted by the perceived strength of semantic associations.

Here, across four experiments, we systematically manipulate the semantic relationship between objects and scenes to directly test the degree to which task-irrelevant semantic associations interact.Experiments 1-3 investigate the influence of semantic relationship between objects and scenes.It is hypothesized that if semantic associations guide attentional selection, independent of task, the high-level information extracted from a scene will facilitate processing of semantically related objects, leading to more efficient processing of targets presented on semantically related task-irrelevant object.Importantly, we further show a direct relationship between the strength of semantic association and the magnitude of attentional facilitation (e.g., a computer mouse in an office would be processed faster than when in a bathroom).Lastly, Experiment 4 investigates how the object-to-object semantic relationship and object-to-scene semantic relationships interact, with the prediction that maximum attentional benefit will be observed when all semantic relationships are congruent, suggesting a more efficient processing of information.

Experiment 1: Task-irrelevant semantic scene-to-object relationship influences attention
Accuracy A two-way repeated measure analysis of variance (ANOVA) was conducted with scene category (office, living room, bathroom, kitchen, bedroom) and semantic relationship (semantically related, SR; semantically not related, NR) as within-subjects variables.Main effects were not significant (Fs < 1.0, ps > 0.36).As hypothesized, there was a significant interaction with scene category and semantic relationship, F(4, 64) = 3.60, p = 0.010, η 2 p = 0.184.In a follow up analysis, consistent with our hypothesis, participants were significantly more accurate (M = 90.17%) in the non-related (NR) than the semantically related (SR) condition (M = 86.83%)for the bedroom scene category (F(1) = 9.30, p = 0.008), and marginally more accurate in the SR (M = 91.54%)than the NR (M = 88.43%)condition for the bathroom category (F(1) = 4.38, p = 0.053).No other significant difference existed for the remaining three scene categories (ps > 0.40).

Semantic regression
Following the significant interaction between scene category and semantic relationship, we directly tested the prediction that the semantic benefit is only as strong as the strength of semantic relationship between objects and scenes.We thus investigated whether attentional facilitation is directly dependent on the strength of association that an object has with the scene.To determine whether the strength of semantic association was predictive of the amount of semantic benefit, data from the survey were utilized.For each scene category, the average rating for all eight objects that were NR (e.g., oven mitt, alarm clock, toothpaste, etc. for office) was subtracted from the rating of a SR object (e.g., mouse for office) to calculate the semantic index, with a greater value indicating a stronger semantic association.This calculation was done for all objects used in the experiment.Then, a linear regression analysis was conducted to test whether the strength of the semantic association significantly predicted the amount of semantic facilitation in RT (Fig. 1B), defined as the difference in RT between the NR and SR condition (NR -SR; greater value indicating greater semantic benefit).
The results indicated that the semantic association was highly predictive of overall semantic facilitation, R 2 adj = 0.54, F(1,8) = 11.67,p = 0.009.For example, objects associated with the bathroom (toothpaste and toilet paper; red data points) were strongly associated with the scene resulting in a strong semantic facilitation, while objects associated with the bedroom (clothes hanger and alarm clock; blue data points) were not strongly associated, leading to a weaker semantic facilitation.
The current results show that attentional allocation is influenced by task-irrelevant meaningful associations between a scene and objects.RT data indicate that participants were significantly faster at target identification when the target appeared on the semantically-related object than the non-related object, linking attentional allocation with semantic relatedness.This semantic effect interacted with scene category, where office and bathroom scenes yielded largest facilitation, with weaker effects in living room, kitchen, and bedroom.To test whether the semantic benefit is directly related to the strength of semantic association of object to the scene, the strength of semantic association between objects and scenes was measured.Using the derived semantic strength, a regression analysis indicated that the strength of semantic association is highly predictive of the amount of semantic facilitation: the more strongly associated an object is with a scene, relative to other objects, the stronger the amount of attentional facilitation.These results indicate that semantic relationships between scenes and objects can be processed and influence attention in an involuntary manner, demonstrating that the influence of task-irrelevant semantic information on attentional allocation can be expanded to object-scene relationships.

Experiment 2: Strength of semantic association predicts amount of semantic influence on attention
Experiment 1 demonstrated that task-irrelevant semantic scene-object relationship influences attentional allocation by facilitating attention towards the semantically related object.To test the generalizability of this effect, multiple exemplars per scene category were utilized, and the Amazon Mechanical Turk platform was used to recruit a more diverse population.Additionally, to directly relate each individual's semantic rating to facilitation of RTs, participants were asked to complete the semantic survey at the end of the experiment.

Accuracy
A two-way repeated measures ANOVA was conducted with scene category (office, living room, bathroom, kitchen, and bedroom) and semantic relationship (SR, NR) as within subjects variable.No main effects or interaction reached significance (Fs < 2.31, ps > 0.06).

Regression analysis
A linear regression analysis was conducted to test whether the strength of the semantic association directly predicted the amount of semantic facilitation in RT.The results indicated that the semantic association was highly predictive of overall semantic facilitation, R 2 adj = 0.19, F(1,128) = 30.69,p < 0.001 (Fig. 1D).To assess the nature of this significant relationship, participants' semantic ratings were linearly regressed against each scene category.This revealed that semantic ratings were strongly predictive of the amount of attentional facilitation within the office category, R 2 adj = 0.14, F(1,24) = 5.20, p = 0.032, the living room category,R 2 adj = 0.38, F(1,24) = 16.02,p < 0.001, and the bedroom category, R 2 adj = 0.19, F(1,24) = 6.80, p = 0.015, but not for the kitchen or bathroom category (Fs < 1.68, ps > 0.20).
Using multiple exemplars, Experiment 2 internally replicated and extended our findings showing that attention is influenced by the scene-object semantic relationship.This semantic benefit again interacted with scene category, such that participants showed the effect in the predicted direction in the office and bathroom scenes.A regression analysis indicated that overall, the strength of semantic association was highly predictive of the amount of semantic facilitation: a stronger object-scene association leads to greater attentional facilitation.This finding partially explains why some scenes did not yields a strong semantic bias (i.e., a weak object-to-scene association yields a weak semantic benefit).

Experiment 3: Semantic facilitation occurs independent of statistical association
Thus far, we show strong evidence for the influence of task-irrelevant semantic scene-object association on attentional allocation.However, despite the fact that scene-object semantic relationship was not informative of the target location, one of two possible semantically-related objects always appeared in any condition (e.g., either a mouse or calculator was always present in the office).Thus, participants may have explicitly, or implicitly, learned to associate specific objects with a scene [21][22][23] resulting in faster preparatory responses to the SR object.To ensure the task-irrelevant nature of the object-scene semantic relationship, a control condition where neither object was related to the scene was added, eliminating any predictive scene-object associations.

Accuracy
A two-way repeated measures ANOVA was conducted with scene category (office, living room, bathroom, kitchen, and bedroom) and semantic relationship (SR, NR, control) as within-subjects variable.No main effects or interaction reached significance (Fs < 2.12, ps > 0.08).

Regression analysis
Only data from the SR and NR conditions were used in the regression analysis to calculate the behavioral semantic facilitation effect consistently with Experiments 1 and 2. A linear regression analysis (identical analysis conducted for previous Experiments) was conducted to test whether the strength of the semantic association significantly predicted the amount of semantic facilitation in RT.The results indicated that the semantic association was highly predictive of overall semantic facilitation, R 2 adj = 0.17, F(1,83) = 18.50, p < 0.001 (Fig. 2B).To assess the nature of this significant relationship, participants' semantic ratings were linearly regressed against each scene In Experiment 3, an overall effect of scene-object semantic relationship on attentional allocation was observed, even when participants were no longer able to predict presence of any particular object in a scene.As in prior experiments, faster performance was observed for targets appearing on semantically-related objects than on non-related or control objects, demonstrating that the semantically-related object is being processed preferentially.Noticeably, there was no significant difference between the NR and control condition, suggesting that the task-irrelevant semantic association between an object and a scene facilitates attentional allocation rather than inhibits non-related information.Lastly, a linear regression analysis replicated the findings of the previous two experiments, such that the strength of semantic relationship predicted the amount of behavioral semantic facilitation across all scenes as well as separately in the office, living room, and bedroom scene categories.Again, there was no relationship between the strength of semantic relationship and behavioral facilitation in the bathroom and kitchen, replicating the findings of the previous experiment.

Experiment 4: Task-irrelevant semantic scene-object relationship interacts with object-object relationship
While Experiments 1-3 established the effect of task-irrelevant semantic relationship between objects and scene on attention, scenes are comprised of multiple objects forming even more relationships with one another 24,25 .Considering that the semantic relationship between objects can influence attention in both task-relevant and irrelevant situations, it is important to understand how the object-to-object relationship interacts with object-toscene relationship to influence attention.In Experiment 4 the degree to which object-object relationship interacts with the semantic relationship between object and scene was examined.We hypothesized that the greatest attentional facilitation would occur when both the object-object and object-scene relationships are related, while the least amount of facilitation would be observed when neither were related.Note that as in prior experiments objects and scenes remain task-irrelevant.

Accuracy
A three-way repeated measures ANOVA with scene category (office, living room, bathroom, kitchen, bedroom), scene relationship (scene SR, scene NR) and object relationship (object SR, object NR) was conducted for accuracy and RT.No interaction or main effect reached significance for accuracy (Fs < 2.44, ps > 0.05).

Semantic regression
Using the strength of semantic association values for all 10 objects obtained in Experiment 1, a linear regression analysis was conducted to test whether the strength of the semantic association significantly predicted the amount of semantic facilitation in RT (Fig. 3C).The results indicated that the semantic association was highly predictive of overall semantic facilitation, R 2 adj = 0.59, F(1,8) = 14.14, p = 0.006.Thus, as seen consistently across previous experiments, objects that were rated as strongly associated with a scene showed the greatest behavioral facilitation while objects that were rated as being weakly associated showed the least facilitation.
Experiment 4 investigated the degree to which object-object semantic relationship interacts with scene-object semantic relationship to understand how different types of high-level associations interact to influence attentional allocation.The results replicated previous findings: participants' performance was facilitated when the target appeared on top of the semantically related object.More importantly, there was an interaction between objectobject relationship and object-scene relationship, such that the greatest facilitation was when both the objects were related to each other as well as the scene.Thus, maximum benefit was seen when everything was related to one another, suggesting a more efficient processing of scene information.

Discussion
When viewing scenes, object and semantic information influences attentional guidance far more than saliency information alone 15,[26][27][28] .While the focus has traditionally been on situations in which semantic information is relevant to the task [7][8][9][10]29 , semantic information is a continuous element of the real-world and thus its influence might extend beyond task-relevance. Acordingly, a growing literature argues that high-level information from real-world scenes is processed automatically and can impinge on cognitive processes 19,20 and guide attention 16,30 .However, a question that remains unaddressed is whether the semantic information of a scene as well its relationship with an object can influence attention.Considering that objects are fundamental to a scene and contribute to scene perception 24,31,32 , it is important to understand how the task-irrelevant semantic relationship between an object and a scene contributes to attentional allocation.
Across four experiments, we demonstrated that task-irrelevant, scene-object semantic associations directly bias attention.Experiments 1-2 showed that a scene's semantic properties bias attention towards the semantically related object and that the amount of this facilitation is directly driven by the strength of object-scene association.Experiment 3 established that the observed attentional benefit is not a result of learned associations, but of task-irrelevant semantic associations.Lastly, Experiment 4 tested the degree to which semantic relationship between an object and a scene interacts with the semantic association between two objects, demonstrating that attentional facilitation was greatest when all relationships were congruent to the scene category.
While there is evidence implicating obligatorily processed semantic information influencing attention, research has largely focused on isolating the semantic relationship, such as examining the relationship between a few objects 16,17 or the effect of overall meaning within a scene 30 .This isolation allows a more controlled investigation, but it also restricts understanding of the dynamics of how various semantic relationships interact in the real-world scenes.For instance, when examining task-irrelevant semantic relationships between two, maximum three, real-world objects 16,17 , attention is biased towards the semantically related object while other studies have shown that inconsistent information may result in longer fixations, but does not necessarily capture attention 33 .Here, we provide evidence that the semantic properties of scenes automatically facilitate attention towards semantically related objects regardless of its task-relevance.This attentional benefit also scaled with the strength of semantic association, with greater behavioral benefit for stronger object-scene semantic associations.Thus, the intrusive influence that semantic information has on attentional allocation does not stop with objects, but can be extended to the semantic association between objects and scene.

Data analysis
All participants with an average accuracy rate of less than 80% were removed from the analysis due to lack of attention.For each participant, all RT less than 200 ms and greater than 1500 ms were removed from the analysis as anticipatory responses and attention lapses respectively (1.12, 0.9, 2.07, and 1.38% of trials were removed in Experiments 1, 2, 3, and 4 respectively).Response time (RT) and accuracy data for each experiment were analyzed using repeated-measures analysis of variance (ANOVA).Only correct trials were used to calculate mean RT for each participant.

Design and procedure
Each trial began with a fixation cross presented on a gray screen for 500 ms.Afterwards, a scene image was presented for 1,000 ms with participants being instructed to maintain fixation.This was followed by the onset of a peripheral object pair, 3.5° from either side of fixation (Fig. 4B) for 750 ms.Critically, one of the objects was always semantically related (SR) to the scene while the other object was not related (NR).After the objects were displayed for 750 ms, two target Gabor patches (one smaller than the other) and distractor was presented.The smaller Gabor patch (0.3°) was always presented in the middle of the scene, on top of the fixation cross.The larger Gabor patch (1.5°) was presented in the middle of one of the two objects and a checkerboard distractor (1.5°) appeared on the other object.The semantic relationship of the object on which the target Gabor appeared defined the experimental condition (e.g., SR = target on a semantically related object, NR = target on non-related object object).The Gabor patches were oriented 45° either to the left or right.Participants were instructed to maintain fixation and report whether the Gabor patches matched in orientation with a key press.Crucially, the larger Gabor patch appeared equally on both the SR and NR object, rendering the scene-object semantic association task-irrelevant.The targets/distractor were presented for 200 ms, after which participants had 2,000 ms to respond.The brief presentation of the targets and distractor as well as the spatial distance between the fixation and objects required participants to maintain fixation to accurately perform the task, controlling for any potential eye movements.Correct trials were separated by a 500 ms intertrial interval (ITI), and incorrect trials were indicated by a red fixation cross and a 1,000 ms ITI.After a brief practice, participants were presented with the experiment.
In Experiment 1, participants performed a total of 640 trials (10 blocks of 64 trials) in which the stimuli appeared in random order.An online survey was conducted using Google Forms (on a separate set of participants), to assess the strength of semantic association between the scene and the corresponding objects used in the experiment.In this survey, participants were presented with an image of an object and a scene (Fig. 4C) and asked to rate how likely the object would appear in the scene on a scale of 1 (not very likely)-6 (very likely).These values were used to calculate the strength of semantic association to determine whether, as predicted, the strength of semantic relationship between an object and a scene can be used to predict the strength of semantic benefit.
In Experiment 2, participants performed a total of 320 trials (5 blocks of 64 trials).The number of trials were halved to ensure participants would focus and finish the experiment within 30 min.Participants also took part in the semantic rating survey at the end of the experiment.Participants were presented with an image of an object and were asked to rate how likely the object would appear in one of the five scene categories on a scale of 1 (not very likely)-6 (very likely).For each scene category, the average rating for all eight objects that were NR for a certain scene was subtracted from the average rating of the SR objects to calculate the semantic index, indicating the strength of the semantic association.Experiment 3 was identical to Experiment 1 except for the new control condition.The control condition consisted of trials in which, for each scene category, two objects from a separate category (e.g., toilet paper and toothpaste for office) were designated as the two objects appearing with NR objects (i.e., neither of the objects in the control condition were semantically related to each other nor the scene).With the addition of the control condition, participants were now equally likely to see trials in which an object was related to the scene and trials in which neither was related.50% of all trials were the control condition with the remaining half equally split between the SR and NR condition.
Experiment 4 utilized a 2 (object-object relationship: object-SR, object-NR) × 2 (object-scene relationship: scene-SR, scene-NR).The condition was defined based on the semantic relationship of the object on which the target appeared (Fig. 3A).Thus, an object-SR condition would mean that the object with the target Gabor patch would be semantically related to the other object and the scene-SR condition would mean that the scene would be semantically related to the object with the target Gabor patch.As with all previous experiments, the target could appear on the objects with equal probability, rendering semantic associations task-irrelevant.All other aspects of the experiment was identical to Experiment 1.

Figure 1 .
Figure 1.(A) RT results of Experiment 1. (B) Regression analysis for Experiment 1 indicated that the strength of semantic association significantly predicted the amount of semantic facilitation.Two data points per category represent the two semantically related objects for each scene.Error bars indicate ± 1 standard error.(C) RT results for Experiment 2. (D) Regression analysis again revealed a significant relationship between semantic index and behavioral semantic facilitation.Datapoints here represent each individual participant for each scene.

Figure 2 .
Figure 2. (A) RT data from Experiment 3. (B) Regression analysis for Experiment 3. Greater semantic association led to greater behavioral facilitation.

Figure 3 .
Figure 3. (A) Example of all 4 possible experimental conditions.From top left in clockwise order, object-SR & scene-SR, object-NR & scene-SR, object-NR & scene-NR, object-SR & Scene-NR (B) RT Results of Experiment 4. (C) Regression analysis for Experiment 4 indicated that the strength of semantic association again significantly predicted the amount of semantic facilitation.

Figure 4 .
Figure 4. (A) Full set of stimuli used in Experiment 1. Objects below each scene category are the two objects designated as semantically related.(B) Experimental paradigm of Experiment 1-Participants' task was to maintain fixation and report whether the orientation of the center Gabor matched the orientation of the Gabor on either one of the objects.Please refer to the Methods section for more details.(C) Example of survey used to rate strength of semantic association between scenes and objects.